Hitherto, for PCI buses with I/O slots or standard I/O buses such as a PCI Express (PCIe), a hot plug has been standardized which allows online insertion and removal of an I/O device (such as a network interface card (NIC)). A hot plug in the past is provided for an I/O slot. When an I/O device is inserted or removed online to or from an I/O slot, a hot plug controller issues an interrupt to an OS. The OS refers to the state of the hot plug controller and determines whether the I/O device has been inserted to the I/O slot or removed from the I/O slot. In accordance with the determination result, the OS enables or disables the corresponding device driver.
FIG. 1 is a schematic configuration diagram of a server and an I/O device. Referring to FIG. 1, a server 100 includes a CPU (Central Processing Unit) 111, a memory 112, a root complex 113, a PCI Express switch (hereinafter, called a PCIe switch) 114 and I/O slots 120 and 121. The memory 112 stores control programs for implementing a virtual machine, including a hypervisor 104, and a BIOS (Basic Input/Output System) 106. The hypervisor 104 may control one or more virtual machines (sometimes called VMs). The hypervisor 104 includes drivers 105A to 105C which control functions 122-0 to 122-3 and 123-0, which will be described below. The CPU 111 executes the hypervisor 104 within the memory 112 to implement virtual machines (VMs) 101, 102, and 103. The CPU 111 reads and executes the BIOS 106 stored in the memory 112. The root complex 113 is a core device of a PCIe system and internally contains a host bridge and is connected to the CPU 111. The PCIe switch 114 is a device for increasing the number of I/O slots to be connected to I/O devices. The I/O slots 120 and 121 are mechanisms for connecting the I/O devices 122 and 123, respectively.
The PCIe switch 114 includes an upstream PCI-to-PCI bridge (hereinafter, called a P2P bridge) 115, and downstream P2P bridges 116 and 118. The CPU 111 side will be called an upstream, and the side having the I/O devices 122 and 123 will be called a downstream. The upstream P2P bridge 115 and downstream P2P bridges 116 and 118 connect an upstream bus (or primary bus), not illustrated, and a downstream bus (or secondary bus), not illustrated, and transfer packets between the two buses. The downstream P2P bridges 116 and 118 include hot plug controllers 117 and 119, respectively. The hot plug controllers 117 and 119 detect the I/O device to be inserted or removed online to or from the I/O slots and use an interrupt signal to notify the hypervisor 104 of the insertion or removal of the I/O device.
The I/O device 122 includes a plurality of functions 122-0 and 122-1. The I/O device 123 also includes a function 123-0. In a PCIe system, the I/O devices 122 and 123 positioned at the most downstream parts may sometimes be called end points. Each of the functions 122-0 and 122-1 is a unit which provides a set of functions within the I/O device 122 to the hypervisor 104 or BIOS 106. The characteristic that a plurality of functions are provided within one I/O device is called multifunction. For example, when the I/O device 122 is a dual port NIC having two ports, the I/O device 122 has two functions. The hypervisor 104 may assign a function to be used to a virtual machine. For example, the hypervisor 104 assigns the function 122-0 to a virtual machine 101 and assigns the function 122-1 to the virtual machine 102.
Under the PCIe standard, the BIOS 106 when the server 100 is started assigns a set of a bus number, a device number and a function number to the functions within the root complex 113, upstream P2P bridge 115, downstream P2P bridges 116 and 118, and I/O devices 122 and 123. For example, the BIOS 106, as in the logically connected state illustrated in FIG. 2, sets a set of a bus number, a device number and a function number to the functions within the root complex 113, upstream P2P bridge 115, downstream P2P bridge 116 and I/O device 122. FIG. 3 is a block diagram illustrating a configuration of the PCIe switch 114.
The PCIe switch 114 includes the upstream P2P bridge 115, downstream P2P bridges 116 and 118, upstream port 130, switch 131, and downstream ports 132 and 133. Notably, the downstream port 132 corresponds to the I/O slot 120 in FIG. 1, and the downstream port 133 corresponds to the I/O slot 121 in FIG. 1.
The upstream port 130 provides a physical link for connecting to the root complex 113 in FIG. 1. The downstream ports 132 and 133 provide physical links for connecting to the I/O devices 122 and 123, respectively, in FIG. 1. The upstream P2P bridge 115 has a configuration table 115A and uses the configuration table 115A to control the transfer of a packet received from the hypervisor 104 or I/O devices 122 and 123. The switch 131 changes the internal destination (which may be the downstream P2P bridge 116 or 118) of a packet.
The downstream P2P bridge 116 includes a bridge control unit 142, a configuration processing unit 143 and a configuration table 144. The bridge control unit 142 refers to the configuration table 144 to control the packet transfer. The configuration table 144 stores information to be referred or set as a configuration space by the hypervisor 104 or BIOS 106. The configuration table 144 stores information on the state of the I/O slot 120 including whether the I/O device 122 has been connected or not. The configuration table 144 stores information for providing a function of the hot plug controller 117 in FIG. 1. The configuration processing unit 143 has a function of the hot plug controller 117.
Next, there will be described an operation by the server 100 when the I/O device 122 is connected to the server 100 and an operation by the server 100 when the I/O device 122 is removed from the server 100. FIG. 4A is a flowchart illustrating operations by the server 100 when the I/O device 122 is connected to the server 100. FIG. 4B is a flowchart illustrating operations by the server 100 when the I/O device 122 is removed from the server 100.
Referring to FIG. 4A, a circuit (not illustrated) on the I/O slot 120 detects the existence of the I/O device 122 and notifies the existence of the I/O device 122 to the hot plug controller 117 (step S201). The hot plug controller 117 updates the information on the state of the I/O slot 120 within the configuration table 144 with the information describing that the I/O device 122 has been connected (step S202). After that, the hot plug controller 117 issues an interrupt signal to the hypervisor 104 (step S203).
The hypervisor 104 refers to the updated information on the state of the I/O slot 120 within the configuration table 144 and detects the existence of the I/O device 122 (step S204). The hypervisor 104 instructs the hot plug controller 117 to enable the I/O slot 120 (step S205). More specifically, the hypervisor 104 rewrites a flag indicating power supply from the I/O slot 120 to the I/O device 122 within the configuration table 144 to “enable”. Thus, the I/O slot 120 supplies power to the I/O device 122.
Next, the hypervisor 104 scans for the bus corresponding to the I/O slot 120 and detects the functions 122-0 and 122-1 within the I/O device 122 (step S206). More specifically, the hypervisor 104 scans the bus number corresponding to the I/O slot 120 downstream of the downstream P2P bridge 116 illustrated in the logically connected state in FIG. 2. Since the bus number corresponding to the I/O slot 120 downstream of the downstream P2P bridge 116 is “3” which is given to the functions, the hypervisor 104 scans for the bus number “3”. Here, under the PCIe standard, one of the device numbers “0 to 31” is assigned to a device, and one of the function numbers “0 to 7” is assigned to a device. The hypervisor 104 therefore attempts access 256 times (=32×8 times) for the bus number “3”. In the example in FIG. 2, the hypervisor 104 receives responses from the functions 122-0 and 122-1 (or (bus number, device number, function number)=(3,0,0), (3,0,1)) and detects the functions 122-0 and 122-1. On the other hand, the hypervisor 104 receives an error status from the I/O device 122 as a response to an access attempt excluding access to those functions.
The hypervisor 104 assigns address spaces of the hypervisor 104 to the functions 122-0 and 122-1 and reads the corresponding drivers 105A and 105B onto the memory 112 (step S207). The drivers 105A and 105B initialize the functions 122-0 and 122-1 (step S208). The operation by the server 100 ends.
Referring to FIG. 4B, a circuit (not illustrated) on the I/O slot 120 detects the removal of the I/O device 122 and notifies the removal of the I/O device 122 to the hot plug controller 117 (step S211). The hot plug controller 117 updates information on the state of the I/O slot 120 within the configuration table 144 with the information describing that the I/O device 122 has been removed (step S212). After that, the hot plug controller 117 issues an interrupt signal to the hypervisor 104 (step S213).
The hypervisor 104 refers to the update information on the state of the I/O slot 120 within the configuration table 144 and detects the removal of the I/O device 122 (step S204). The hypervisor 104 instructs the hot plug controller 117 to disable the I/O slot 120 (step S215). More specifically, the hypervisor 104 rewrites a flag indicating power supply from the I/O slot 120 to the I/O device 122 within the configuration table 144 to “disable”. Thus, the I/O slot 120 no longer supplies power to the I/O device 122. After that, the hypervisor 104 deletes the drivers 105A and 105B corresponding to the functions 122-0 and 122-1 present on the memory 112 (step S216). The operation by the server 100 ends.
In this way, when the I/O device 122 is connected to the server 100, the hypervisor 104 instructs the hot plug controller 117 to enable the I/O slot 120, and power is supplied to the I/O device 122. As the result, the functions 122-0 and 122-1 within the I/O device 122 are collectively enabled. When the I/O device 122 is removed from the server 100, the hypervisor 104 instructs the hot plug controller 117 to disable the I/O slot 120. The power to be supplied to the I/O device 122 is blocked. As the result, the functions 122-0 and 122-1 within the I/O device 122 are collectively disabled.
Hitherto, a computer system has been known which includes a plurality of virtual computers constructed by a control program provided in a computer and having independent OSs (Operating Systems) running thereon and an I/O device having a single port which is connected to a PCI (Peripheral Component Interconnect) bus of the computer. Another technology has been known in which one set of functions is constructed from an arbitrary combination of a plurality of PCI agents and the functions are controlled from a BIOS (Basic Input/Output System) and/or a driver as one unit.
Japanese Laid-open Patent Publication Nos. 2004-252591 and 9-237246 are examples of related art.
In the server 100, the functions 122-0 and 122-1 within the I/O device 122 are collectively enabled or disabled. It may be difficult for the hypervisor 104 to enable a function within the I/O device 122 independently even when a manager requests to enable the function 122-0 only for assigning the function 122-0 to the virtual machine 101, for example.